Background subtraction (BGS) refers to the ability to remove unwanted background from a live video. Some current video conferencing programs use BGS technology to subtract and replace the background with another prerecorded still or moving background.
There have been several methods developed for BGS using color information only. These methods are either not robust for challenging, but common, situations such as a moving background and changing lighting, or too computationally expensive to be able to run in real-time. The recent emergence of depth cameras provides an opportunity to develop robust, real-time BGS systems using depth information. However, due to current hardware limitations, some of which are fundamental, recorded depth video has poor quality. Notable problems with recorded depth are noisy and instable depth values around object boundaries, and the loss of depth values in hair of a person or shiny object areas, such as belt buckles. As a result, background removal by a simple depth thresholding—referred to as Basic BGS herein—inherits a lot of annoying visual artifacts. Ideally, a robust system will detect and eliminate such visual artifacts, and reduce jitter and roughness around edges contiguous with a removed background.